Papers with NLG evaluation
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing evaluation methods for natural language generation rely on token-level or embedding-level comparisons with text references. |
| Approach: | They propose to use text-to-image generator to generate an image as the embodied imagination for the text snippet and compute the imagination similarity using contextual embeddings. |
| Outcome: | The proposed metric improves existing evaluation metrics’ correlations with human similarity judgments in both reference-based and reference-free scenarios. |
Fusion-Eval: Integrating Assistant Evaluators with LLMs (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent studies have employed large language models (LLMs) as reference-free metrics for NLG evaluation, enhancing adaptability to new tasks tasks. |
| Approach: | They propose a method that leverages large language models to integrate insights from various assistant evaluators. |
| Outcome: | The proposed approach achieves a 0.962 system-level Kendall-Tau correlation with humans on SummEval and a 0.7444 turn-level Spearman correlation on TopicalChat, which is significantly higher than baseline methods. |
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications (2022.naacl-main)
Copied to clipboard
| Challenge: | Evaluating natural language generation systems is difficult, as there are many ways to express similar things in text. |
| Approach: | They combine interviews with NLG practitioners to examine ethical considerations and their implications for NLG evaluation. |
| Outcome: | The findings of the study surface goals, community practices, assumptions, and constraints that shape NLG evaluations, and examine their implications and how they embody ethical considerations. |
On the Intractability to Synthesize Factual Inconsistencies in Summarization (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for detecting factual inconsistencies in abstractive summarization are lacking in factual consistency detection. |
| Approach: | They propose to use real model-generated summaries with human annotations to detect factual inconsistencies. |
| Outcome: | The proposed model outperforms the SOTA on CoGenSumm, FactCC, Frank, and SummEval datasets. |
DEBATE: Devil’s Advocate-Based Assessment and Text Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for evaluating the quality of machine-generated texts have a relatively low correlation with human performance. |
| Approach: | They propose an NLG evaluation framework based on multi-agent scoring system augmented with a concept of Devil’s Advocate. |
| Outcome: | The proposed evaluation framework outperforms the previous state-of-the-art methods in two meta-evaluation benchmarks in NLG evaluation, SummEval and TopicalChat. |
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment (2023.emnlp-main)
Copied to clipboard
| Challenge: | Conventional reference-based metrics have low correlation with human judgments, especially for open-ended generation tasks. |
| Approach: | They propose to use large language models as reference-free NLG evaluators to assess the quality of NLG outputs. |
| Outcome: | The proposed framework outperforms all previous methods in two generation tasks, and has a Spearman correlation of 0.514 with human on summarization task, and a large variance in human judgments. |
A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (2021.eacl-main)
Copied to clipboard
| Challenge: | a lack of transparency is a key issue for robotics and AI. |
| Approach: | They propose to map existing automatic evaluation methods for natural language generation onto explanations. |
| Outcome: | The proposed model shows that embedding-based evaluation methods have higher correlations with human ratings than word-overlap metrics. |
NLG-Metricverse: An End-to-End Library for Evaluating Natural Language Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Natural language generation models are a key component of deep learning, says aaron eliott . he says it is crucial to develop and apply better metrics for NLG evaluation . |
| Approach: | a new open-source library for NLG evaluation is created to facilitate researchers to judge the effectiveness of their models. the framework provides a living collection of NLG metrics in a unified and easy-to-use environment. |
| Outcome: | a new open-source library for NLG evaluation aims to improve performance of models . the framework provides tools to apply, analyze, compare, and visualize the metrics . |
The Authenticity Gap in Human Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using the standard protocol to evaluate NLGs is often violated, resulting in annotator ratings cease to reflect their preferences. |
| Approach: | They propose a human evaluation protocol called system-level probabilistic assessment (SPA) this protocol is based on the assumption that annotators are biased by likert scales . |
| Outcome: | The proposed protocol can recover the ordering of GPT-3 models by size, but less than half of the expected preferences can be recovered when human evaluation is done with the standard protocol. |
Are LLM-based Evaluators Confusing NLG Quality Criteria? (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that LLMs confuse evaluation criteria, which reduces their reliability. |
| Approach: | They propose a hierarchical classification system for 11 common aspects with corresponding different evaluation criteria. |
| Outcome: | The proposed system is based on 11 common aspects with different evaluation criteria. |
Language Model Augmented Relevance Score (2021.acl-long)
Copied to clipboard
| Challenge: | Existing metrics that compare the candidate with the human reference do not consider the context, resulting in poor correlation with human judgements. |
| Approach: | They propose a language model-aware metric that augments the human reference while considering the context to provide evaluation scores that correlate highly with human judgements. |
| Outcome: | The proposed metric achieves higher correlation with human reference judgements and differentiates well-formed candidates from adversarial samples to a larger degree. |
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021.acl-long)
Copied to clipboard
| Challenge: | evaluators distinguish between human- and machine-authored text in three domains without training . evals' accuracy improved up to 55%, but it did not significantly improve across the three domain. |
| Approach: | They examine the role untrained human evaluations play in NLG evaluation and propose ways to improve their evaluations. |
| Outcome: | The evaluators distinguished between human- and machine-authored text at random chance level without training, but their accuracy did not improve across the three domains. |
One Prompt To Rule Them All: LLMs for Opinion Summary Evaluation (2024.acl-long)
Copied to clipboard
Tejpalsingh Siledar, Swaroop Nath, Sankara Muddu, Rupasai Rangaraju, Swaprava Nath, Pushpak Bhattacharyya, Suman Banerjee, Amey Patil, Sudhanshu Singh, Muthusamy Chelliah, Nikesh Garera
| Challenge: | Existing evaluation methods for opinion summarizations lack adequate opinion summary evaluation datasets. |
| Approach: | They propose a dataset that combines 7 dimensions crucial to opinion summaries . they propose OP-I-PROMPT, a dimension-independent prompt, and OP PROMPTS, . |
| Outcome: | The proposed model achieves a Spearman correlation of 0.70 with human judgments, surpassing prior methods. |
Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluation of natural language generation tasks lack reliable data. |
| Approach: | They propose to use annotations from human and GPT-4 to construct a corpus for NLG evaluation. |
| Outcome: | The proposed corpus can perform flexible and interpretable evaluations without references and surpasses existing models. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
CourtEval: A Courtroom-Based Multi-Agent Evaluation Framework (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing automated evaluation metrics like ROUGE and BLEU show low correlation with human judgments. |
| Approach: | They propose a multi-agent evaluation framework that integrates multiple agents . they use ROUGE and BLEU to evaluate natural language models . |
| Outcome: | The proposed evaluation framework outperforms the current state-of-the-art methods in two meta-evaluation benchmarks. |